Image processing device and imaging device

ABSTRACT

Provided is an image processing device including: a 3D-NR unit that performs, using first image data and second image data obtained by capturing images at temporally different times, 3D noise reduction (NR) processing for reducing noise in the first image data; an edge detection unit that detects an edge strength in an image indicated by the 3D-NR processed image data; and a synthesization unit that determines, based on the detected edge strength obtained by the edge detection unit, a synthesis ratio of the first image data and the 3D-NR processed image data, synthesizes the first image data and the 3D-NR processed image data, using the determined synthesis ratio, and outputs synthesized image data obtained by synthesizing the first image data and the 3D-NR processed image data.

CROSS REFERENCE TO RELATED APPLICATION

The present application is based on and claims priority of JapanesePatent Application No. 2012-019483 filed on Feb. 1, 2012. The entiredisclosure of the above-identified application, including thespecification, drawings and claims is incorporated herein by referencein its entirety.

FIELD

The present disclosure relates to image processing devices, and, inparticular, to an image processing device which can performthree-dimensional noise reduction (3D-NR) processing on an inputtedimage.

BACKGROUND

PTL 1 discloses an image processing device which can perform the 3D-NRprocessing on an inputted image. The image processing device can outputan image obtained by performing sharpening processing on edge portionsin the inputted image and performing the 3D-NR processing on imageregions other than the edge portions in the inputted image.

This allows the image processing device to reduce random noise in theimage regions other than the edge portions in the inputted image.

CITATION LIST Patent Literature

[PTL 1] Japanese Unexamined Patent Application Publication No.2011-65339

SUMMARY Technical Problem

However, when the 3D-NR processing is performed in a manner as theconventional image processing device described above performs the 3D-NRprocessing on, for example, an image having a small edge strength suchas an image obtained by capturing a moving subject, there arises aproblem that residual image is rather prominent in the processed image.

One non-limiting and exemplary embodiment provides an image processingdevice which performs the 3D-NR processing on an inputted image andoutputs a resultant image in consideration of the above describedconventional problems, and allows suppression of the occurrence ofresidual image in an outputted image.

Solution to Problem

To solve the above problem, the image processing device according to oneaspect of the present disclosure is an image processing deviceincluding: an image processor that: (i) performs, using first image dataand second image data obtained by capturing images at temporallydifferent times, three-dimensional noise reduction (3D-NR) processingfor reducing noise in the first image data; (ii) detects an edgestrength in an image indicated by the 3D-NR processed image data; (iii)determines, based on the detected edge strength, a synthesis ratio ofthe first image data and the 3D-NR processed image data; and (iv)synthesizes the first image data and the 3D-NR processed image data,using the determined synthesis ratio and outputs synthesized image dataobtained by synthesizing the first image data and the 3D-NR processedimage data.

These general and specific aspects may be implemented using a system, amethod, an integrated circuit, a computer program, or acomputer-readable recording medium such as a CD-ROM, or any combinationof systems, methods, integrated circuits, computer programs, orcomputer-readable recording media.

Advantageous Effects

According to the present disclosure, the image processing device can beprovided which performs the 3D-NR processing on an inputted image,outputs a resultant image, and allows suppression of the occurrence ofresidual image in an outputted image. Moreover, an imaging deviceincluding the image processing device can be provided.

BRIEF DESCRIPTION OF DRAWINGS

These and other objects, advantages and features of the invention willbecome apparent from the following description thereof taken inconjunction with the accompanying drawings that illustrate a specificembodiment of the present disclosure.

FIG. 1 is a block diagram of an example of configuration of a digitalvideo camera.

FIG. 2 is a diagram showing an example of a functional configuration ofa noise reduction unit.

FIG. 3 is a flowchart illustrating processing flow of the noisereduction unit.

FIG. 4 is a diagram showing an example of a relationship between an edgestrength and a coefficient β.

FIG. 5 is a diagram showing an example of a 3D-NR processed image when asubject is stationary.

FIG. 6 is a diagram showing a first example of a 3D-NR processed imagewhen a subject is moving.

FIG. 7 is a diagram showing a second example of a 3D-NR processed imagewhen a subject is moving.

FIG. 8 is a diagram showing an example of a result of processingperformed by the noise reduction unit.

FIG. 9 is a diagram showing various examples of a relationship betweenthe edge strength in YOUT and a proportion (β) of YIN indicated in asynthesis ratio.

FIG. 10 is a diagram illustrating the relationship between the edgestrength and β when a proportion of a rate of change in β to a rate ofchange in edge strength dynamically changes.

DESCRIPTION OF EMBODIMENT

In relation to the conventional techniques, the inventors have found thefollowing problems:

For example, in a plurality of images which are taken by a static cameraand in which a moving object is captured, the object is blurred and thusthe edge strength in the images at which the object is displayed has arelatively small value.

Thus, the conventional image processing device performs the 3D-NRprocessing on the image of the object. In other words, processing occurswhich synthesizes a plurality of images having residual images addedthereto and in which the position of the object is different. As aresult, there arises a problem that the 3D-NR processed image showsstronger residual images than the image before the 3D-NR processing isperformed thereon.

To solve such a problem, an image processing device according to oneaspect of the present disclosure is an image processing deviceincluding: an image processor that: (i) performs, using first image dataand second image data obtained by capturing images at temporallydifferent times, three-dimensional noise reduction (3D-NR) processingfor reducing noise in the first image data; (ii) detects an edgestrength in an image indicated by the 3D-NR processed image data; (iii)determines, based on the detected edge strength, a synthesis ratio ofthe first image data and the 3D-NR processed image data; and (iv)synthesizes the first image data and the 3D-NR processed image data,using the determined synthesis ratio and outputs synthesized image dataobtained by synthesizing the first image data and the 3D-NR processedimage data.

According to the above configuration, the first image data and theprocessed image data that is obtained by performing the 3D-NR processingon the first image data are synthesized using the synthesis ratiodetermined by the synthesization unit.

Specifically, the synthesis ratio is determined based on the edgestrength in an image indicated by the processed image data. Thus, forexample, when the edge strength is relatively small, such as the casewhere the image indicated by the processed image data is a portion ofthe residual image of the subject, controls is possible such as reducingthe proportion of the processed image data in the synthesization. As aresult, the occurrence of residual image in an image to be outputted issuppressed.

Moreover, for example, when the detected edge strength indicates asecond edge strength greater than a first edge strength, the imageprocessor may determine a proportion of the 3D-NR processed image dataindicated in the synthesis ratio to have a greater value than aproportion when the detected edge strength indicates the first edgestrength.

According to the above configuration, for example, in each case wherethe edge strength is relatively large and where the edge strength isrelatively small, the suppression of residual image caused by the 3D-NRprocessing and reduction of noise by the 3D-NR processing are achievedin a balanced manner.

Moreover, for example, the image processor may determine the synthesisratio to have a greater proportion of the 3D-NR processed image dataindicated in the synthesis ratio as the edge strength indicated in thedetected edge strength increases.

According to the above configuration, the suppression of residual imagecaused by the 3D-NR processing and reduction of noise by the 3D-NRprocessing are achieved in a balanced manner, in accordance with, forexample, the edge strength that can vary.

Moreover, for example, when a brightness indicated in the 3D-NRprocessed image data is a second brightness lower than a firstbrightness, the image processor may determine the proportion of the3D-NR processed image data in the synthesis ratio that corresponds tothe detected edge strength to have a greater value than the proportionwhen the brightness is the first brightness.

According to the above configuration, for example, even when the outlineof the subject is not sharp due to dark image-capturing environment, thesynthesized image data in which noise in the outline portion is reducedby the 3D-NR processing is outputted.

Moreover, for example, the first image data may be image data in apredetermined unit that is included in one of two frames obtained bycapturing the images at temporally different times and the second imagedata may be image data in the predetermined unit that is included in theother of the two frames, and the image processor may, for eachpredetermined unit, determine the synthesis ratio and synthesize thefirst image data and the 3D-NR processed image data to outputsynthesized image data corresponding to the one of the two frames.

According to the above configuration, for example, image data whichmakes up a picture and in which at least one of the suppression ofresidual image and the reduction of noise is appropriately performed foreach region is outputted as the synthesized image data.

Moreover, for example, the image processor may perform the 3D-NRprocessing, using processed image data that is the second image data andhas been outputted from the image processor before the 3D-NR processingis performed on the first image data.

According to the above configuration, the image processor can use imagedata already obtained by previous 3D-NR processing to perform the 3D-NRprocessing on the first image data which is a current target to beprocessed. As a result, for example, effects of suppressing theoccurrence of noise in the synthesized image data increase.

Moreover, an imaging device according to one aspect of the presentdisclosure includes: the image processing device according to any ofaspects described above; and an image sensor that captures images toobtain the first image data and the second image data.

According to the above configuration, an imaging device can be achievedwhich reduces noise, using the 3D-NR processing and allows suppressionof the occurrence of residual image in an image to be outputted orstored.

These general and specific aspects may be implemented using a system, amethod, an integrated circuit, a computer program, or acomputer-readable recording medium such as a CD-ROM, or any combinationof systems, methods, integrated circuits, computer programs, orcomputer-readable recording media.

Embodiment

Hereinafter, an image processing device according to an embodiment willbe described with reference to the accompanying drawings. It should benoted that figures are schematic views and do not necessarily illustratethe present disclosure precisely.

It should be noted that the embodiment described below is merely apreferred illustration of the present disclosure. Values, shapes,materials, components, disposition or a form of connection between thecomponents, steps, and the order of the steps are merely illustrative,and are not intended to limit the present disclosure. Moreover, amongcomponents of the below non-limiting embodiments, components not setforth in the independent claims indicating the top level concept of thepresent disclosure will be described as optional components.

1-1. Configuration

First, a configuration of a digital video camera 100 according to thepresent embodiment will be described with reference to FIG. 1.

FIG. 1 is a block diagram showing an example of the configuration of thedigital video camera 100 according to the embodiment.

The digital video camera 100 includes an imaging unit 101 and an imageprocessing device 160.

The imaging unit 101 includes an optical system 110, a lens drive unit120, a diaphragm 300, a shutter 130, and a CMOS (Complementary MetalOxide Semiconductor) image sensor 140.

In the imaging unit 101, the CMOS image sensor 140 captures a subjectimage formed by the optical system 110 which includes one or more lens.

Image data generated by the CMOS image sensor 140 undergoes variousprocessing by the image processing device 160, and is stored in a memorycard 200. Hereinafter, the configuration of the digital video camera 100will be described in detail.

The optical system 110 includes a zoom lens, a focus lens, and the like.Moving the zoom lens along with the optical axis can enlarge and reducethe subject image. Moving the focus lens along with the optical axis canadjust the focus of the subject image.

The lens drive unit 120 drives movement of various lens included in theoptical system 110. The lens drive unit 120 includes, for example, azoom motor which drives the zoom lens and a focus motor which drives thefocus lens.

The diaphragm 300 adjusts the size of aperture manually or in accordancewith user settings, to adjust the amount of light passing through theaperture of the diaphragm 300.

The shutter 130 blocks light reaching the CMOS image sensor 140 via theshutter 130.

The CMOS image sensor 140 captures the subject image formed by theoptical system 110 to generate image data. The CMOS image sensor 140performs various operations of exposure, transfer, an electronicshutter, and the like.

An analog-to-digital converter 150 converts analog image data generatedby the CMOS image sensor 140 into digital image data.

The image processing device 160 performs various processing on the imagedata (more specifically, referring to digital image data that hasundergone conversion by the analog-to-digital converter 150,hereinafter,) generated by the CMOS image sensor 140 to generate imagedata to be displayed on a display monitor 220, generate image data to bestored in the memory card 200, and the like.

The image processing device 160 performs various processing on the imagedata generated by the CMOS image sensor 140, such as gamma correction,white balance correction, and the defect correction.

Moreover, the image processing device 160 compresses the image datagenerated by the CMOS image sensor 140 in a compression format complyingwith the H.264 standard or the MPEG2 standard.

The image processing device 160 according to the present embodimentincludes a noise reduction unit 161, as a characteristic functionalconfiguration, which performs processing, including the 3D-NRprocessing, on the image data to reduce noise and residual images. Thenoise reduction unit 161 will be described in detail below, withreference to FIG. 2.

It should be noted that the image processing device 160 can be achievedusing digital signal processor (DSP), a microcomputer, or the like.

A controller 180 controls the entirety of the digital video camera 100.The controller 180 can be achieved using a semiconductor device, or thelike. Alternatively, the controller 180 may be achieved using hardwareor a combination of hardware and software. The controller 180 can beachieved using the microcomputer or the like.

A buffer 170 serves as a work memory of the image processing device 160and the controller 180. The buffer 170 can be implemented as, forexample, DRAM or a ferroelectric memory.

A card slot 190 is a device to/from which the memory card 200 can beinserted/removed. Specifically, the card slot 190 is mechanically andelectrically connectable with the memory card 200.

The memory card 200 includes a flash memory, a ferroelectric memory, orthe like therein and can store the image data generated by the imageprocessing device 160, and the like.

An internal memory 240 is a flash memory, a ferroelectric memory, or thelike. The internal memory 240 stores a control program or the like forcontrolling the entirety of the digital video camera 100.

An operation unit 210 is a user interface for receiving operations froma user. The operation unit 210 includes, for example, a cursor pad, adecision button, and the like for receiving operations from a user.

The display monitor 220 can display an image (a through image) indicatedby the image data generated by the CMOS image sensor 140 and an imageindicated by the image data read out from the memory card 200. Inaddition, the display monitor 220 can display various menu screenswhereby the various settings of the digital video camera 100 are made.

1-2. Noise Reduction Unit Included in Image Processing Device

FIG. 2 is a diagram showing an example of a functional configuration ofthe noise reduction unit 161 included in the image processing device 160according to the embodiment.

The noise reduction unit 161 included in the image processing device 160according to the present embodiment will be described with reference toFIG. 2.

A 3D-NR unit 201 is by way of example of an image processing unit forperforming the 3D-NR processing. In the 3D-NR unit 201, calculationshown in Equation 1 is performed as the 3D-NR processing to reduce, fromimage data (referred to as YIN and YMEM) corresponding to inputted twoframes, a noise component of YIN. As a result of the calculation, YOUTby way of example of the processed image data is outputted from the3D-NR unit 201.

YIN is by way of example of first image data. In the present embodiment,YIN is image data obtained by the analog-to-digital converter 150converting the image data generated by the CMOS image sensor 140 intodigital image data. It should be noted that the image processing device160 may perform predetermined processing such as the gamma correction onYIN prior to being inputted to the 3D-NR unit 201 and a synthesizationunit 203.

YMEM is by way of example of second image data. YMEM is image data to beread out from the buffer 170 in the present embodiment. In other words,YMEM is image data that has been outputted from the 3D-NR unit 201before the 3D-NR processing is performed on YIN, and stored in thebuffer 170. In other words, YMEM is past YOUT.

YOUT=YIN−α (YIN−YMEM)   Eq. 1

where α is any value greater than or equal to 0 and less than or equalto 1, and is a variable whereby the strength of the noise reductioneffect is determined.

Next, the principle of how noise is reduced by Equation 1 will bedescribed. The case is assumed where YIN contains noise and YMEM isnoiseless.

In this case, YOUT approximates to YMEM by subtracting the component α(YIN−YMEM) from YIN. As a result, the noise component is reduced.

For example, the case is assumed where an original image data value ofYIN=100, the noise component added to YIN=20, and an image data value ofYMEM=100 (assuming noiseless).

In this case, when α=0.5, YOUT=120−0.5 (120−100)=110 is satisfied. Thus,the noise component can be reduced.

The synthesization unit 203 synthesizes the image data (YIN and YOUT)corresponding to the inputted two frames, using a synthesis ratio(β:1−β) determined by the synthesization unit 203 as shown in Equation2.

In other words, YIN (the first image data) and YOUT (the processed imagedata) are synthesized in the ratio of β:1−β.

In other words, as a value of β increases, a proportion of YIN in YMIXto be outputted from the synthesization unit 203 increases and aproportion of YOUT in YMIX decreases. It should be noted that YMIX is byway of example of the synthesized image data.

YMIX=βYIN+(1−β) YOUT   Eq. 2

The edge detection unit 204 detects the edge strength in the imageindicated by the inputted image data YOUT.

Specifically, the edge detection unit 204 extracts an edge componentwhich is included in the image indicated by the inputted image dataYOUT. In extraction of the edge component, for example, the convolutionoperation for an edge detection filter (such as Sobel filter orLaplacian filter) is performed on the image data YOUT, and, as a resultof the calculation, the edge strength in the image data is outputtedfrom the edge detection unit 204.

The synthesization unit 203 determines the synthesis ratio (β:1−β),based on the detection result obtained by the edge detection unit 204.

1-3. Data Flow

Data and processing flow of the noise reduction unit 161 included in theimage processing device 160 according to the present embodiment will bedescribed with reference to FIG. 2 and FIG. 3.

FIG. 3 is a flowchart illustrating an example of the processing flow ofthe noise reduction unit 161 included in the image processing device 160according to the present embodiment.

The image data YIN inputted from the analog-to-digital converter 150 tothe image processing device 160 and the image data YMEM outputted fromthe buffer 170 are inputted to the 3D-NR unit 201.

The 3D-NR unit 201 performs the noise reduction processing (the 3D-NRprocessing) on the image data YIN by making the calculation shown inEquation 1 (S10). YOUT, which is the processed image data obtained bythe 3D-NR processing, is inputted to the synthesization unit 203, theedge detection unit 204, and the buffer 170.

The edge detection unit 204 detects the edge strength in an imageindicated by the image data YOUT inputted from the 3D-NR unit 201 (S20).The detection result is inputted to the synthesization unit 203.

The synthesization unit 203 determines the synthesis ratio (β:1−β) inEquation 2, in accordance with the edge strength indicated in thedetection result inputted from the edge detection unit 204 (S30).

It should be noted that the edge detection unit 204 determines, forexample, β greater than or equal to 0 and less than and equal to 1,thereby determining the synthesis ratio (β:1−β). In other words, β and1−β are variables in a relationship that when one of β and 1−β isdetermined, the other is determined. Thus, a fact that one of β and 1−βis determined means that the synthesis ratio (β:1−β) is determined.

Using the determined synthesis ratio, the synthesization unit 203synthesizes the image data YIN and the image data YOUT which is inputtedfrom the 3D-NR unit 201 (S40).

The synthesization unit 203 outputs image data YMIX obtained bysynthesizing the image data YIN and the image data YOUT (S50). In otherwords, the image data YMIX is handled, by the digital video camera 100,as image data outputted from the noise reduction unit 161 included inthe image processing device 160.

It should be noted that, specifically, in the present embodiment, theseries of processing steps shown in FIG. 3 is performed in predeterminedunits such as every pixel and every block of pixels.

For example, if the predetermined unit is a block of n pixels×m pixels(where n and m are integers greater than or equal to 2), an average orsum of the edge strength each calculated for each pixel, or the edgestrength in a portion of the pixels included in the block is handled asthe edge strength corresponding to the block.

In other words, the 3D-NR processing (Equation 1) and the synthesizationprocess (Equation 2) are performed on one frame to be processed inpredetermined units, and, as a result, the synthesized image data (YMIX)corresponding to the frame is obtained.

It should be noted that the predetermined units in which the 3D-NRprocessing is performed and the predetermined units in which thesynthesization processing is performed may not match.

Here, in the present embodiment, the relationship between the edgestrength and β satisfies the relationship shown in FIG. 4, for example.Specifically, the synthesization unit 203 stores synthesis ratioinformation which is a function or table indicating the relationshipbetween the edge strength and β shown in FIG. 4, for example. In otherwords, the synthesization unit 203 can calculate β that corresponds tothe edge strength inputted from the edge detection unit 204, based onthe synthesis ratio information and the edge strength.

In other words, in the present embodiment, the synthesization unit 203calculates β so that the greater the edge strength detected by the edgedetection unit 204 is, the smaller β is.

This means that in synthesization of YIN and YOUT, the greater the edgestrength in the image indicated by YOUT is, the greater the proportionof YOUT (1−β) indicated in the synthesis ratio of YIN and YOUT is. Thereason why such processing is performed will be described below.

1-4. Reason Why Residual Image Reduces

The image processing device 160 according to the present embodiment canreduce the residual image which occurs due to the 3D-NR processing, byperforming the processing according to the above-mentioned data flow,which will be described with reference to FIG. 5 to FIG. 8.

FIG. 5 is a diagram showing an example of a 3D-NR processed image when asubject is stationary.

FIG. 6 is a diagram showing a first example of a 3D-NR processed imagewhen a subject is moving.

FIG. 7 is a diagram showing a second example of a 3D-NR processed imagewhen a subject is moving.

As shown in FIG. 5, the case is assumed where the 3D-NR processing (theprocessing described in section 1-2) is performed to reduce noise in theimage 402, using an image 401 and an image 402 obtained by capturingimages of the stationary subject (an apple) at temporally differenttimes.

In other words, the case is assumed where an image corresponding to YMEMis the image 401 and an image corresponding to YIN is the image 402.

In this case, an image 403 obtained by performing the 3D-NR processingusing the image 401 and the image 402 has reduced noise, and has noresidual image of the subject in the image that is caused by the 3D-NRprocessing. In other words, a sharp image of the apple is obtained.

Meanwhile, as shown in FIG. 6, the case is assumed where the 3D-NRprocessing is performed to reduce noise in the image 502, using an image501 and an image 502 which are obtained by capturing, at temporallydifferent times, images of a scene in which some subject (apple) ismoving.

In other words, the case is assumed where an image corresponding to YMEMis the image 501 and an image corresponding to YIN is the image 502.

In this case, in the image 503 obtained by performing the 3D-NRprocessing using the image 501 and the image 502, although noise isreduced, the residual image caused by the 3D-NR processing ends upoccurred in the periphery of the subject the image of which is capturedwhen the subject is moving.

This is because when the image of the moving subject is captured, aposition of the subject in an image indicated by YIN ends up displacedfrom a position of the subject in an image indicated by YMEM. In otherwords, this is because the component of YIN ends up outputted at aspatial location at which the subject is originally not present in theimage indicated by YMEM. As a result, the component of YIN appears asresidual image in the image 503.

More specifically, as shown in FIG. 7, an image captured of a movingsubject such as an image 601 and an image 602, has a low edge strength.

In addition, performing the 3D-NR processing using images having lowedge strengths such as the image 601 and the image 602 results ingenerating an image such as an image 603 in which residual image isappeared to a greater extent as compared to the images before the 3D-NRprocessing is performed using the same.

As can be seen from the above, the relationship between the movement ofthe subject and the edge strength is strong. Specifically, if the imageis captured when the subject is moving, the edge strength correspondingto the subject in the image decreases, and if the image is captured whenthe subject is stationary, the edge strength corresponding to thesubject in the image increases.

Thus, in the above processing flow according to the present embodiment,the image data YOUT and the image data YIN are synthesized by changing,based on the edge strength in the image indicated by the 3D-NR processedimage data YOUT, the synthesis ratio of the 3D-NR processed image dataYOUT and the image data YIN before the 3D-NR processing. This reducesthe volume of residual image occurred.

FIG. 8 is a diagram showing an example of a result of the processingperformed by the noise reduction unit 161 included in the imageprocessing device 160 according to the embodiment.

As described above, performing the 3D-NR processing on the image 502,using two images (501 and 502) captured of the scene which includes themoving subject (apple) results in occurrence of the residual image inthe periphery of the subject in the image 503 which is the result of the3D-NR processing.

However, the noise reduction unit 161 according to the presentembodiment synthesizes the image 502 to be made to undergo the 3D-NRprocessing and the image 503 which is the result of the 3D-NRprocessing, by changing the synthesis ratio depending on a region in theobtained image so that residual image is reduced.

For example, taking image data of a region a which is a portion of theresidual image as an example, the image data has a relatively low edgestrength in an image of the region a. Thus, a relatively large value isemployed as β. In other words, a relatively small value is employed as aproportion (1−β) of the image 503 indicated in the synthesis ratio (seeEquation 2 and FIG. 4).

As a result, the region a has a high proportion of the component of theimage 502 (corresponding to YIN) which is the image before the 3D-NRprocessing and in which no residual image caused by the 3D-NR processingis included. Thus, the residual image in an image 504 indicated by YMIX,which is the synthesized image data, is reduced to a greater extent thanthe volume of the residual image in the image 503 which is the result ofthe 3D-NR processing.

For example, taking image data of a region b which is a portion of thestationary subject as an example, the image data has a relatively highedge strength in an image of the region b. Thus, a relatively smallvalue is employed as β indicated in the synthesis ratio. In other words,a relatively large value is employed as a proportion (1−β) of the image503 (see Equation 2 and FIG. 4).

As a result, the region b has a high proportion of the component of theimage 503 (corresponding to YMEM) which is an image the noise therein isreduced by the 3D-NR processing. This reduces noise in the image 504indicated by YMIX which is the synthesized image data.

As described above, the noise reduction unit 161 according to thepresent embodiment can suppress the occurrence of residual image whichis caused by the 3D-NR processing in an image, while reducing noise inthe image by the 3D-NR processing.

2. Complement to Embodiment

The embodiment according to the present disclosure is described above.The present disclosure, however, is not limited to the above embodiment.Thus, some complement to the embodiment will be described.

In the above embodiment, the CMOS image sensor 140 is illustrated by wayof example of imaging means included in the imaging unit 101. However,the present disclosure is not limited thereto. For example, the imagingmeans may be implemented by a charge coupled device (CCD) image sensoror a negative channel metal oxide semiconductor (NMOS) image sensor.

Moreover, the image processing device 160 and the controller 180 may beimplemented using a semiconductor chip or separate semiconductor chips.

Moreover, the 3D-NR processing performed by the 3D-NR unit 201 is notlimited to a particular method. While YOUT (past YOUT) stored in thebuffer 170 is used as YMEM for the subsequent 3D-NR processing (see FIG.2) in the present embodiment, the 3D-NR unit 201 may use, for example,two of YIN which are continuously obtained, instead of YMEM, and performthe 3D-NR processing to reduce noise in one of YIN.

Moreover, the relationship between the edge strength and 8 may not be inproportional to each other as shown in FIG. 4.

FIG. 9 is a diagram showing various examples of the relationship betweenthe edge strength in YOUT and the proportion (β) of YIN indicated in thesynthesis ratio used for synthesizing YIN and YOUT.

For example, as the edge strength increases, β may decrease in astepwise manner as shown in (a) of FIG. 9, instead of continuouslydecreasing.

Moreover, as shown in (b) of FIG. 9, β may be either one of two stepvalues “0” and “1”. For example, if the edge strength is less than orequal to a threshold E, β may=1, and if the edge strength is greaterthan the threshold E, β may=0.

In other words, briefly, if the edge strength is less than or equal tothe threshold E, the synthesization unit 203 may output only theinputted image data (YIN) as the synthesized image data, and, if theedge strength is greater than the threshold E, the synthesization unit203 may output only the 3D-NR processed image data (YOUT) as thesynthesized image data (see Equation 2).

As described above, limiting the possible values of β to a plurality ofstep values reduces synthesization processing load in the synthesizationunit 203, for example.

It should be noted that when β is either “0” or “1”, either YIN or YOUTis employed as the synthesized image data (YMIX) in units of thesynthesization processing by the synthesization unit 203. Thus, it isconceivable that a difference in image quality between YIN and YOUT maybe prominent at a boundary between a region where YIN is employed and aregion where YOUT is employed in YMIX corresponding to one frame.

However, employing a relatively small unit such as pixel to pixel, as aunit of the synthesization processing, allows the difference in imagequality between YIN and YOUT, which is visually perceivable in YMIX, tobe kept to a substantially insignificant extent.

Moreover, the relationship between the edge strength and β is notnecessarily represented by a linear function (a linear line) in atwo-dimensional plane formed by an edge strength axis and a β axis. Forexample, the relationship between the edge strength and β may berepresented by a curve.

For example, as shown in (c) of FIG. 9, the greater the edge strengthis, the greater the reduction rate of β (a decreasing value of β perunit length in the horizontal direction) may be.

In this case, β has a relatively large value while, for example, theedge strength is about an intermediate level. Thus, the proportion ofYIN component is greater than the proportion of YOUT component in YMIXwhich is the result of synthesization of YIN and YOUT. In other words,the suppressive effect on the occurrence of residual image caused by the3D-NR processing in YMIX improves.

Moreover, for example, as shown in (d) of FIG. 9, the greater the edgestrength is, the lower the reduction rate of β may be.

In this case, β has a relatively small value while, for example, theedge strength is about an intermediate level. Thus, the proportion ofthe component of YOUT is greater than the proportion of the component ofYIN in YMIX which is the result of synthesization of YIN and YOUT. Inother words, the effects by the 3D-NR processing to reduce noise to bepresent in YMIX improves.

As described above, the relationship between the edge strength and β maybe any insofar as being negatively correlated to each other, forexample. In other words, β according to the edge strength may bedetermined so that the edge strength and the proportion of YOUT (1−β)indicated in the synthesis ratio (β:1−β) is positively correlated.

In other words, if the detection result obtained by the edge detectionunit 204 indicates a second edge strength greater than a first edgestrength, the synthesization unit 203 may determine the synthesis ratioso that the proportion of YOUT (1−β) indicated in the synthesis ratio(β:1−β) is greater than that when the detection result indicates thefirst edge strength.

Moreover, the proportion of the rate of change in β to the rate ofchange in edge strength (for example, the slope of the edge strengthversus β shown in FIG. 4) may dynamically change according to theimage-capturing environment.

FIG. 10 is a diagram illustrating the relationship between the edgestrength and β when the proportion of the rate of change in β to therate of change in edge strength dynamically changes.

The following cases are assumed: a case where, for example, theimage-capturing environment for the digital video camera 100 isenvironment A that is relatively bright such as the outdoor in fineweather; and a case where the image-capturing environment is environmentB that is relatively dark such as the outdoor in cloudy weather.

In this case, in general, image data having a high brightness isobtained as the result of capturing an image in the environment A, andimage data having a low brightness is obtained as the result ofcapturing an image in the environment B.

In other words, in general, the outline of a subject shown in the imageobtained in the environment B is not as sharp as the outline of thesubject shown in the image obtained in the environment A.

Thus, the lower the brightness of YOUT, from which the edge strength isto be detected, is, the lower the slope of the edge strength versus β isset (the angle of negative slope is increased) as shown in (a) and (b)of FIG. 10, for example.

In other words, when the brightness indicated in YOUT is a secondbrightness lower than a first brightness, the synthesization unit 203may determine the synthesis ratio so that the proportion of YOUT in thesynthesis ratio corresponding to the edge strength indicated by thedetection result obtained by the edge detection unit 204 has a greatervalue than that when the brightness is the first brightness.

Because of this, even when the outline of the subject is not sharp, thesynthesized image data (YMIX) in which noise in the outline portion isreduced by the 3D-NR processing is outputted.

For example, the case is assumed where due to a fact that theenvironment for capturing an image of the subject is dark, the edgestrength in the outline portion of a certain subject has E1 which is arelatively small value.

In this case, when the relationship between the edge strength and β isas shown in (a) of FIG. 10, β1 which corresponds to E1 and is arelatively large value is calculated as the proportion of YIN indicatedin the synthesis ratio used for synthesizing YIN and YOUT.

On the other hand, when the relationship between the edge strength and βis as shown in (b) of FIG. 10, β2 smaller than β1 is calculated as theproportion of YIN corresponding to E1.

In other words, the proportion of YOUT (1−β) which is the 3D-NRprocessed image data has a larger value in the synthesis ratio withrespect to the outline portion of the subject in the case of (b) shownin FIG. 10 than the case of (a) shown in FIG. 10.

Thus, the image of the outline portion in which noise is reduced by the3D-NR processing can be obtained as the synthesization result, even whenthe outline of the subject is not sharp in YOUT because of, for example,relatively dark image-capturing environment.

Briefly, the lower the brightness of YOUT is, the lower the slope of theedge strength versus β is set (the angle of negative slope isincreased), thereby increasing the sensitivity in recognizing theoutline of the subject. As a result, with respect to the image capturedin relatively dark environment, reduction of noise by the 3D-NRprocessing and suppression of the occurrence of residual image caused bythe 3D-NR processing are achieved in a balanced manner.

It should be noted that the brightness of YOUT may be identified usingone or more pixel values of YOUT, or, using one or more pixel values ofYIN or YMEM.

Moreover, instead of using the pixel values of YOUT and the like, thebrightness of YOUT may be identified using, for example, informationacquired from a device external to the image processing device 160, suchas setting values of the diaphragm 300 included in the digital videocamera 100 or brightness information from a sensor which detects thebrightness of the image-capturing environment.

As described above, the digital video camera 100 according to thepresent embodiment includes the image processing device 160 and, usingthe noise reduction unit 161, the image processing device 160 can reducenoise by the 3D-NR processing and reduce residual image caused by the3D-NR processing.

It should be noted that each of the components in the embodiment maytake the form as dedicated hardware or may be implemented by executing asoftware program suitable for each component. Each component may beimplemented by, CPU (Central Processing Unit) or a program executionunit, such as processor, loading and executing a software program storedin a hard disk or a recording medium such as a semiconductor memory.Here, the software program for implementing the image processing deviceaccording to the embodiment is as follows.

In other words, the program causes a computer to execute the followingimage processing method: an image processing method including:performing, using first image data and second image data which areobtained by capturing images at temporally different times,three-dimensional noise reduction (3D-NR) processing for reducing noisein the first image data; detecting an edge strength in an imageindicated by processed image data which is image data on which the 3D-NRprocessing has been performed; determining, based on a result of thedetection, a synthesis ratio of the first image data and the processedimage data; synthesizing the first image data and the processed imagedata, using the determined synthesis ratio; and outputting synthesizedimage data obtained by the synthesization.

While the image processing device according to only one or moreexemplary embodiments of the present disclosure has been described basedon the exemplary embodiment, the present disclosure is not limited tothe exemplary embodiment. Various modifications to the presentembodiments that may be conceived by those skilled in the art andcombinations of components of different embodiments are intended to beincluded within the scope of the one or more exemplary embodiments,without departing from the spirit of the one or more exemplaryembodiments.

For example, the noise reduction unit 161 may be included as an imageprocessing device in a device other than the digital video camera 100.

For example, the noise reduction unit 161 may be included as an imageprocessing device for reducing noise and residual image in a mobileterminal, a stationary device, or the like that has imagingfunctionality.

Moreover, for example, the present disclosure may be achieved as animage processing device which performs the series of processing steps(for example, see FIG. 3) on image data to be outputted from a digitalvideo camera that does not include the noise reduction unit 161.

Although only some exemplary embodiments of the present disclosure havebeen described in detail above, those skilled in the art will readilyappreciate that many modifications are possible in the exemplaryembodiments without materially departing from the novel teachings andadvantages of the present disclosure. Accordingly, all suchmodifications are intended to be included within the scope of thepresent disclosure.

INDUSTRIAL APPLICABILITY

The present disclosure is applicable to image processing devices forreducing noise and residual image in imaging devices such as digitalvideo cameras, mobile phone cameras, and smartphones.

1. An image processing device comprising: an image processor that: (i)performs, using first image data and second image data obtained bycapturing images at temporally different times, three-dimensional noisereduction (3D-NR) processing for reducing noise in the first image data;(ii) detects an edge strength in an image indicated by the 3D-NRprocessed image data; (iii) determines, based on the detected edgestrength, a synthesis ratio of the first image data and the 3D-NRprocessed image data; and (iv) synthesizes the first image data and the3D-NR processed image data, using the determined synthesis ratio andoutputs synthesized image data obtained by synthesizing the first imagedata and the 3D-NR processed image data.
 2. The image processing deviceof claim 1, wherein, when the detected edge strength indicates a secondedge strength greater than a first edge strength, the image processordetermines a proportion of the 3D-NR processed image data indicated inthe synthesis ratio to have a greater value than a proportion when thedetected edge strength indicates the first edge strength.
 3. The imageprocessing device of claim 1, wherein the image processor determines thesynthesis ratio to have a greater proportion of the 3D-NR processedimage data indicated in the synthesis ratio as the edge strengthindicated in the detected edge strength increases.
 4. The imageprocessing device of claim 2, wherein, when a brightness indicated inthe 3D-NR processed image data is a second brightness lower than a firstbrightness, the image processor determines the proportion of the 3D-NRprocessed image data in the synthesis ratio that corresponds to thedetected edge strength to have a greater value than the proportion whenthe brightness is the first brightness.
 5. The image processing deviceof claim 1, wherein the first image data is image data in apredetermined unit that is included in one of two frames obtained bycapturing the images at temporally different times and the second imagedata is image data in the predetermined unit that is included in theother of the two frames, and the image processor, for each predeterminedunit, determines the synthesis ratio and synthesizes the first imagedata and the 3D-NR processed image data to output synthesized image datacorresponding to the one of the two frames.
 6. The image processingdevice of claim 1, wherein the image processor performs the 3D-NRprocessing, using processed image data that is the second image data andhas been outputted from the image processor before the 3D-NR processingis performed on the first image data.
 7. An imaging device comprising:the image processing device of claim 1; and an image sensor thatcaptures images to obtain the first image data and the second imagedata.
 8. An integrated circuit comprising: an image processor that: (i)performs, using first image data and second image data obtained bycapturing images at temporally different times, three-dimensional noisereduction (3D-NR) processing for reducing noise in the first image data;(ii) detects an edge strength in an image indicated by the 3D-NRprocessed image data; (iii) determines, based on the detected edgestrength, a synthesis ratio of the first image data and the 3D-NRprocessed image data; and (iv) synthesizes the first image data and the3D-NR processed image data, using the determined synthesis ratio andoutputs synthesized image data obtained by synthesizing the first imagedata and the 3D-NR processed image data.
 9. An image processing methodcomprising: performing, using first image data and second image dataobtained by capturing images at temporally different times,three-dimensional noise reduction (3D-NR) processing for reducing noisein the first image data; detecting an edge strength in an imageindicated by the 3D-NR processed image data; determining, based on thedetected edge strength, a synthesis ratio of the first image data andthe 3D-NR processed image data; synthesizing the first image data andthe 3D-NR processed image data, using the determined synthesis ratio;and outputting synthesized image data obtained by the synthesization.